The present invention relates to distributed database systems and methods for implementing asynchronous replication with automatic failover and/or automatic reintegration of failed systems. Conventional database architectures are designed to provide for reliable processing of database transactions, such as read and write operations performed on the database. Different database architecture designs stress different aspects of the well-known ACID properties (atomicity, consistency, isolation, durability), and such designs typically provide trade-offs between the properties and/or processing concerns based on which properties are stressed. As the demand for processing power and speed has increased, data stores have been developed to provide scaling of databases across a number of systems, vastly increasing the capability of handling large volumes of requests. Ecommerce websites, in particular, have vast need of quick and reliable processing of transactions. Moreover, such websites typically need to be capable of processing transactions even while the systems that host the data store are failing or losing connectivity with the service hosting the website.
In typical distributed database systems, no single node is responsible for all data affecting a transaction. Distribution of responsibility introduces significant additional complexity to ensure such ACID properties are fulfilled by the database design. Communication failures, failures of individual nodes, and inconsistent transactions all present additional complexity in reliable processing of database transactions. Some conventional approaches resolve some of these reliability issues by separating responsibility for read and write operations among elements of a distributed database. For example, master-slave relationships can be established between nodes in a distributed database. The well-known MySQL database is configured for master-slave replication of database instances. In the MySQL database, processing of write requests can be limited to the master system which can then propagate changes to its slave systems. The slave systems are then used to respond to read requests, permitting a large volume of read operations to occur across an easily scalable number of systems. Known trade-offs occur in such a setting, for example write processing capability is limited. Master-slave architectures are best suited for settings that require large volumes of read operations and a smaller number of write operations.
Other systems provide for multiple nodes within a data store that can process write requests, replicate their data to other nodes, and respond to read requests. For example, the well-known Amazon Dynamo database system provides an architecture that employs keys to determine a set of nodes that can process writes and reads for particular data. The Dynamo system emphasizes availability of data over other concerns, including consistency. By always permitting write operations whenever possible, conflicting transactions can and do arise. Thus, the Dynamo system requires implementation of reconciliation logic for conflicting transactions in the database and may require vector clocks, for example, to associate timing information with various updates. The timing information can then be processed to resolve conflicting operations.